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Abstract 

This paper describes a method for estimat- 
ing conditional probability distributions over 
the parses of "unification-based" grammars 
which can utilize auxiliary distributions that 
are estimated by other means. We show how 
this can be used to incorporate information 
about lexical selectional preferences gathered 
from other sources into Stochastic "Unification- 
based" Grammars (SUBGs). While we ap- 
ply this estimator to a Stochastic Lexical- 
Functional Grammar, the method is general, 
and should be applicable to stochastic versions 
of HPSGs, categorial grammars and transfor- 
mational grammars. 

1 Introduction 

"Unification-based" Grammars (UBGs) can 
capture a wide variety of linguistically impor- 
tant syntactic and semantic constraints. How- 
ever, because these constraints can be non-local 
or context-sensitive, developing stochastic ver- 
sions of UBGs and associated estimation pro- 
cedures is not as straight-forward as it is for, 
e.g., PCFGs. Recent work has shown how to 
define probability distributions over the parses 
of UBGs ( Abney, 1997 ) and efficiently estimate 
and use conditional probabilities for parsing 
( Johnson et al., 1999| ). Like most other practical 
stochastic grammar estimation procedures, this 
latter estimation procedure requires a parsed 
training corpus. 

Unfortunately, large parsed UBG corpora are 
not yet available. This restricts the kinds of 
models one can realistically expect to be able 
to estimate. For example, a model incorporat- 
ing lexical selectional preferences of the kind 
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described below might have tens or hundreds 
of thousands of parameters, which one could 
not reasonably attempt to estimate from a cor- 
pus with on the order of a thousand clauses. 
However, statistical models of lexical selec- 
tional preferences can be estimated from very 
large corpora based on simpler syntactic struc- 
tures, e.g., those produced by a shallow parser. 
While there is undoubtedly disagreement be- 
tween these simple syntactic structures and the 
syntactic structures produced by the UBG, one 
might hope that they are close enough for lexical 
information gathered from the simpler syntactic 
structures to be of use in defining a probability 
distribution over the UBG's structures. 

In the estimation procedure described here, 
we call the probability distribution estimated 
from the larger, simpler corpus an auxiliary dis- 
tribution. Our treatment of auxiliary distribu- 
tions is inspired by the treatment of reference 
distributions in Jelinek's ( |1997]) presentation of 
Maximum Entropy estimation, but in our es- 
timation procedure we simply regard the loga- 
rithm of each auxiliary distribution as another 
(real-valued) feature. Despite its simplicity, our 
approach seems to offer several advantages over 
the reference distribution approach. First, it 
is straight-forward to utilize several auxiliary 
distributions simultaneously: each is treated as 
a distinct feature. Second, each auxiliary dis- 
tribution is associated with a parameter which 
scales its contribution to the final distribution. 
In applications such as ours where the auxiliary 
distribution may be of questionable relevance 
to the distribution we are trying to estimate, it 
seems reasonable to permit the estimation pro- 
cedure to discount or even ignore the auxiliary 
distribution. Finally, note that neither Jelinek's 
nor our estimation procedures require that an 
auxiliary or reference distribution Q be a prob- 



ability distribution; i.e., it is not necessary that 
Q{^}) = 1, where 17 is the set of well-formed 
linguistic structures. 

The rest of this paper is structured as fol- 
lows. Section ^ reviews how exponential mod- 
els can be defined over the parses of UBGs, 
gives a brief description of Stochastic Lexical- 
Functional Grammar, and reviews why maxi- 
mum pseudo-likelihood estimation is both feasi- 
ble and sufficient of parsing purposes. Section ^ 
presents our new estimator, and shows how it 
is related to the minimization of the Kullback- 
Leibler divergence between the conditional es- 
timated and auxiliary distributions. Section ^ 
describes the auxiliary distribution used in our 
experiments, and section ^ presents the results 
of those experiments. 

2 Stochastic Unification-based 
Grammars 

Most of the classes of probabilistic language 
models used in computational linguistic are ex- 
ponential families. That is, the probability P(u;) 
of a well-formed syntactic structure u; E is de- 
fined by a function of the form 



where /(w) G is a vector of feature values, 
A E El™" is a vector of adjustable feature param- 
eters, Q is a function of uj (which Jelinek ( |1997] ) 
calls a reference distribution when it is not an in- 
dicator function), and Zx = Q{uj)e^'^^^^ doj is 
a normalization factor called the partition func- 
tion. (Note that a feature here is just a real- 
valued function of a syntactic structure uj; to 
avoid confusion we use the term "attribute" to 
refer to a feature in a feature structure). If 
Q{ijj) = 1 then the class of exponential dis- 
tributions is precisely the class of distributions 
with maximum entropy satisfying the constraint 
that the expected values of the features is a cer- 
tain specified value (e.g., a value estimated from 
training data), so exponential models are some- 
times also called "Maximum Entropy" models. 

For example, the class of distributions ob- 
tained by varying the parameters of a PCFG 
is an exponential family. In a PCFG each rule 
or production is associated with a feature, so ni 
is the number of rules and the jth feature value 
fj{ijj) is the number of times the j rule is used 



in the derivation of the tree u; E 17. Simple ma- 
nipulations show that Pa('^) is equivalent to the 
PCFG distribution if Xj = \ogpj, where is the 
rule emission probability, and Q{uj) = Zx = \. 

If the features satisfy suitable Markovian in- 
dependence constraints, estimation from fully 
observed training data is straight-forward. For 
example, because the rule features of a PCFG 
meet "context-free" Markovian independence 
conditions, the well-known "relative frequency" 
estimator for PCFGs both maximizes the likeli- 
hood of the training data (and hence is asymp- 
totically consistent and efficient) and minimizes 
the Kullback-Leibler divergence between train- 
ing and estimated distributions. 

However, the situation changes dramatically 
if we enforce non-local or context-sensitive con- 
straints on linguistic structures of the kind that 
can be expressed by a UBG. As Abney ( |1997| ) 
showed, under these circumstances the relative 
frequency estimator is in general inconsistent, 
even if one restricts attention to rule features. 
Consequently, maximum likelihood estimation 
is much more complicated, as discussed in sec- 
tion 2.2. Moreover, while rule features are natu- 



ral for PCFGs given their context-free indepen- 
dence properties, there is no particular reason 
to use only rule features in Stochastic UBGs 
(SUBGs). Thus an SUBG is a triple (G,/,A), 
where G is a UBG which generates a set of well- 
formed linguistic structures 17, and / and A are 
vectors of feature functions and feature param- 
eters as above. The probability of a structure 
a; E 17 is given by (||) with Q{uj) = 1. Given a 
base UBG, there are usually infinitely many dif- 
ferent ways of selecting the features / to make 
a SUBG, and each of these makes an empirical 
claim about the class of possible distributions 
of structures. 

2.1 Stochastic Lexical Functional 
Grammar 

Stochastic Lexical-Functional Grammar 
(SLFG) is a stochastic extension of Lexical- 
Functional Grammar (LFG), a UBG formalism 
developed by Kaplan and Bresnan ( |1982| ). 
Given a base LFG, an SLFG is constructed 
by defining features which identify salient 
constructions in a linguistic structure (in LFG 
this is a c-structure/f-structure pair and its 
associated mapping; see Kaplan ( |1995|) ). Apart 
from the auxiliary distributions, we based our 



features on those used in Johnson et al. ( |199S| ), 
which should be consulted for further details. 
Most of these feature values range over the 
natural numbers, counting the number of times 
that a particular construction appears in a 
linguistic structure. For example, adjunct and 
argument features count the number of adjunct 
and argument attachments, permitting SLFG 
to capture a general argument attachment pref- 
erence, while more specialized features count 
the number of attachments to each grammatical 
function (e.g., SUBJ, OBJ, COMP, etc.). 
The flexibility of features in stochastic UBGs 
permits us to include features for relatively 
complex constructions, such as date expres- 
sions (it seems that date interpretations, if 
possible, are usually preferred), right-branching 
constituent structures (usually preferred) and 
non-parallel coordinate structures (usually 
dispreferred). Johnson et al. remark that they 
would have liked to have included features for 
lexical selectional preferences. While such fea- 
tures are perfectly acceptable in a SLFG, they 
felt that their corpora were so small that the 
large number of lexical dependency parameters 
could not be accurately estimated. The present 
paper proposes a method to address this by 
using an auxiliary distribution estimated from 
a corpus large enough to (hopefully) provide 
reliable estimates for these parameters. 

2.2 Estimating stochastic 

unification-based grammars 

Suppose uj = LiJi, . . . , u;„ is a corpus of n syn- 
tactic structures. Letting /j(u3) = ^27=1 fji^i) 
and assuming each G O, the likelihood of the 
corpus L;s^(u;) is: 



La(^) 
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^X-m ^-n (2) 
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log La (^) = /, (J))-nEA(/, ) (3) 



where Ex{fj) is the expected value of fj un- 
der the distribution P^. The maximum likeli- 
hood estimates are the A which maximize (^) , or 
equivalently, which make (^ zero, but as John- 
son et al. ( 1999D explain, there seems to be no 
practical way of computing these for realistic 
SUBGs since evaluating (^ and its derivatives 



dll) involves integrating over all syntactic struc- 
tures fi. 

However, Johnson et al. observe that parsing 
applications require only the conditional prob- 
ability distribution Fx{uj\y), where y is the ter- 
minal string or yield being parsed, and that this 
can be estimated by maximizing the pseudo- 
likelihood of the corpus PLa(u)): 



PLa(cD) = l[Px{io^\y^) 

i=l 

n 

= e^-/(^) l[Zx\y^) (4) 

i=l 

In (^) , yi is the yield of uji and 



Zxiyi 



.UJ, 



where i^{yi) is the set of all syntactic structures 
in Q with yield yi (i.e., all parses of yi gener- 
ated by the base UBG). It turns out that cal- 
culating the pseudo-likelihood of a corpus only 
involves integrations over the sets of parses of 
its yields 0(yi), which is feasible for many inter- 
esting UBGs. Moreover, the maximum pseudo- 
likelihood estimator is asymptotically consistent 
for the conditional distribution F{c o\y). For the 
reasons explained in Johnson et al. ( |1999| ) we ac- 
tually estimate A by maximizing a regularized 
version of the log pseudo-likelihood (P), where 
fjj is 7 times the maximum value of fj found in 
the training corpus: 



logPLA(tl') 



m 

j=i '^"j 



^2 (5) 



See Johnson et al. (|1999D for details of the calcu- 
lation of this quantity and its derivatives, and 
the conjugate gradient routine used to calcu- 
late the A which maximize the regularized log 
pseudo-likelihood of the training corpus. 

3 Auxiliary distributions 

We modify the estimation problem presented in 



section by assuming that in addition to the 
corpus UJ and the m feature functions / we are 
given k auxiliary distributions Qi, . . . , whose 
support includes O that we suspect may be re- 
lated to the joint distribution P(a;) or condi- 
tional distribution P(a;|?/) that we wish to esti- 



mate. We do not require that the Qj be proba- 
bihty distributions, i.e., it is not necessary that 
jQQj{uj)du) = 1, but we do require that they 
are strictly positive (i.e., Qjioj) > 0,Va; € Q). 
We define k new features fm+i, ■ ■ ■ , fm+k where 
fm+ji}^) = logQj(ti;), which we call auxiliary 
features. The m + k parameters associated with 
the resulting m+k features can be estimated us- 
ing any method for estimating the parameters 
of an exponential family with real-valued fea- 
tures (in our experiments we used the pseudo- 
likelihood estimation procedure reviewed in sec- 
tion 2.2). Such a procedure estimates parame- 
ters Xm+ii ■ ■ ■ , Am+fc associated with the auxil- 
iary features, so the estimated distributions take 
the form (|6|) (for simplicity we only discuss joint 
distributions here, but the treatment of condi- 
tional distributions is parallel). 



Pa(^) 



■m+j 



(6) 



Note that the auxiliary distributions Qj are 
treated as fixed distributions for the purposes 
of this estimation, even though each Qj may it- 
self be a complex model obtained via a previous 
estimation process. Comparing (|6|) with (|^) on 
page |2|, we see that the two equations become 
identical if the reference distribution Q in is 
replaced by a geometric mixture of the auxiliary 
distributions Qj, i.e., if: 



Q{uj) 



The parameter associated with an auxiliary fea- 
ture represents the weight of that feature in the 



mixture. If a parameter A, 



m+j 



1 then the 



corresponding auxiliary feature Qj is equivalent 
to a reference distribution in Jelinek's sense, 
while if Xm+j = then Qj is effectively ig- 
nored. Thus our approach can be regarded as 
a smoothed version Jelinek's reference distribu- 
tion approach, generalized to permit multiple 
auxiliary distributions. 

4 Lexical selectional preferences 

The auxiliary distribution we used here is based 
on the probabilistic model of lexical selectional 
preferences described in Rooth et al. ( 1999 ). An 
existing broad-coverage parser was used to find 



shallow parses (compared to the LFG parses) 
for the 117 million word British National Cor- 
pus (Carroll and Rooth, 1998). We based our 



auxiliary distribution on 3.7 million {g, r, a) tu- 
ples (belonging to 600,000 types) we extracted 
these parses, where g is a lexical governor (for 
the shallow parses, g is either a verb or a prepo- 
sition) , a is the head of one of its NP arguments 
and r is the the grammatical relationship be- 
tween the governor and argument (in the shal- 
low parses r is always OBJ for prepositional gov- 
ernors, and r is either subj or OBJ for verbal 
governors). 

In order to avoid sparse data problems we 
smoothed this distribution over tuples as de- 
scribed in (iRooth et al., 19991 ). We assume thai, 
governor-relation pairs {g, r) and arguments a 
are independently generated from 25 hidden 
classes C, i.e.: 



P((5,r,a)) 



cec 



Pe((<?,r)|c)Pe(a|c)Pe(c) 



where the distributions Pg are estimated from 
the training tuples using the Expectation- 
Maximization algorithm. While the hidden 
classes are not given any prior interpretation 
they often cluster semantically coherent pred- 
icates and arguments, as shown in Figure |l|. 
The smoothing power of a clustering model such 
as this can be calculated explicitly as the per- 
centage of possible tuples which are assigned a 
non-zero probability. For the 25-class model 
we get a smoothing power of 99%, compared 
to only 1.7% using the empirical distribution of 
the training data. 

5 Empirical evaluation 

Hadar Shemtov and Ron Kaplan at Xerox PARC 
provided us with two LFG parsed corpora called 
the Verbmobil corpus and the Homecentre cor- 
pus. These contain parse forests for each sen- 
tence (packed according to scheme described in 
Maxwell and Kaplan ( p.995| )), together with a 
manual annotation as to which parse is cor- 
rect. The Verbmobil corpus contains 540 sen- 
tences relating to appointment planning, while 
the Homecentre corpus contains 980 sentences 
from Xerox documentation on their "homecen- 
tre" multifunction devices. Xerox did not pro- 
vide us with the base LFGs for intellectual prop- 
erty reasons, but from inspection of the parses 
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0.3183 
0.0405 
0.0345 
0.0276 
0.0214 
0.0193 
0.0147 
0.0144 
0.0137 
0.0130 
0.0128 
0.0104 
0.0D9B 
0.0094 
0.0094 
0.0089 
0.0084 
0.0082 
0.0078 
0.0071 
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0.0067 
0.0066 
0.0061 
0.0051 
0.0050 
0.0050 
0.0049 
0.0049 
0.0047 
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say :s 
say:o 
ask:s 
tcll:s 
bc:s 
know : s 
havc:s 
nod:s 
think:s 
shake: s 
take:s 
rcply:s 
smilc:s 
dors 
laugh :s 
tcllro 
saw:s 
add:s 
feel : s 
make : s 
give:s 
ask:o 
shrug :s 
explain:s 
likc:s 
look:s 
sigh:s 
wat ch : s 
hear:s 
answer :s 



Figure 1: A depiction of the highest probabihty predicates and arguments in Class 16. The class 
matrix shows at the top the 30 most probable nouns in the Pe(a|16) distribution and their probabil- 
ities, and at the left the 30 most probable verbs and prepositions listed according to Pre{{g,r)\lG) 
and their probabilities. Dots in the matrix indicate that the respective pair was seen in the training 
data. Predicates with suffix : s indicate the subject slot of an intransitive or transitive verb; the 
suffix : o specifies the nouns in the corresponding row as objects of verbs or prepositions. 



it seems that slightly different grammars were 
used with each corpus, so we did not merge the 
corpora. We chose the features of our SLFG 
based solely on the basis of the Verbmobil cor- 
pus, so the Homecentre corpus can be regarded 
as a held-out evaluation corpus. 

We discarded the unambiguous sentences in 
each corpus for both training and testing (as 
explained in Johnson et al. (|1999 ), pseudo- 
likelihood estimation ignores unambiguous sen- 
tences), leaving us with a corpus of 324 am- 
biguous sentences in the Verbmobil corpus and 
481 sentences in the Homecentre corpus; these 
sentences had a total of 3,245 and 3,169 parses 
respectively. 

The (non-auxiliary) features used in were 
based on those described by Johnson et 
al. ( |1999 ). Different numbers of features 
were used with the two corpora because 
some of the features were generated semi- 



automatically (e.g., we introduced a feature for 
every attribute-value pair found in any feature 
structure), and "pseudo-constant" features (i.e., 
features whose values never differ on the parses 
of the same sentence) are discarded. We used 
172 features in the SLFG for the Verbmobil cor- 
pus and 186 features in the SLFG for the Home- 
centre corpus. 

We used three additional auxiliary features 
derived from the lexical selectional preference 
model described in section ^. These were de- 
fined in the following way. For each governing 
predicate g, grammatical relation r and argu- 
ment a, let n(^g^r,a){^) be the number of times 
that the f-structure: 



PRED = g 
r = [PRED 



appears as a subgraph of the f-structure of 
w, i.e., the number of times that a fills the 



grammatical role r of g. We used the lexical 
model described in the last section to estimate 
P{a\g,r), and defined our first auxiliary feature 
as: 

fl{uj) = logP(5ro)+ ^{g,r,a)(.^)'^ogF{a\g,r 

where go is the predicate of the root feature 
structure. The justification for this feature is 
that if f-structures were in fact a tree, fi{uj) 
would be the (logarithm of) a probability dis- 
tribution over them. The auxiliary feature fi 
is defective in many ways. Because LFG f- 
structures are DAGs with reentrancies rather 
than trees we double count certain arguments, 
so // is certainly not the logarithm of a prob- 
ability distribution (which is why we stressed 
that our approach does not require an auxiliary 
distribution to be a distribution). 

The number of governor-argument tuples 
found in different parses of the same sentence 
can vary markedly. Since the conditional prob- 
abilities P{a\g,r) are usually very small, we 
found that fi{u!) was strongly related to the 
number of tuples found in w, so the parse with 
the smaller number of tuples usually obtains the 
higher /; score. We tried to address this by 
adding two additional features. We set /c(w) to 
be the number of tuples in uj, i.e.: 

Then we set /„(a;) = fi{uj)/ fdio), i.e., /^((j) is 
the average log probability of a lexical depen- 
dency tuple under the auxiliary lexical distribu- 
tion. We performed our experiments with // as 
the sole auxiliary distribution, and with //, fc 
and fn as three auxiliary distributions. 

Because our corpora were so small, we trained 
and tested these models using a 10-fold cross- 
validation paradigm; the cumulative results are 
shown in Table ||. On each fold we evaluated 
each model in two ways. The correct parses 
measure simply counts the number of test sen- 
tences for which the estimated model assigns 
its maximum parse probability to the correct 
parse, with ties broken randomly. The pseudo- 
likelihood measure is the pseudo-likelihood of 
test set parses; i.e., the conditional probability 
of the test parses given their yields. We actu- 
ally report the negative log of this measure, so a 



smaller score corresponds to better performance 
here. The correct parses measure is most closely 
related to parser performance, but the pseudo- 
likelihood measure is more closely related to the 
quantity we are optimizing and may be more 
relevant to applications where the parser has to 
return a certainty factor associated with each 
parse. 

Table |l| also provides the number of indistin- 
guishable sentences under each model. A sen- 
tence y is indistinguishable with respect to fea- 
tures / iff f{uJc) = /("^Oi where is the correct 
parse of y and lJc ^ io' G ^{y), i.e., the feature 
values of correct parse of y are identical to the 
feature values of some other parse of y. If a 
sentence is indistinguishable it is not possible 
to assign its correct parse a (conditional) prob- 
ability higher than the (conditional) probability 
assigned to other parses, so all else being equal 
we would expect a SUBG with with fewer indis- 
tinguishable sentences to perform better than 
one with more. 

Adding auxiliary features reduced the already 
low number of indistinguishable sentences in the 
Verbmobil corpus by only 11%, while it reduced 
the number of indistinguishable sentences in the 
Homecentre corpus by 24%. This probably re- 
flects the fact that the feature set was designed 
by inspecting only the Verbmobil corpus. 

We must admit disappointment with these 
results. Adding auxiliary lexical features im- 
proves the correct parses measure only slightly, 
and degrades rather than improves performance 
on the pseudo-likelihood measure. Perhaps this 
is due to the fact that adding auxiliary features 
increases the dimensionality of the feature vec- 
tor /, so the pseudo-likelihood scores with dif- 
ferent numbers of features are not strictly com- 
parable. 

The small improvement in the correct parses 
measure is typical of the improvement we might 
expect to achieve by adding a "good" non- 
auxiliary feature, but given the importance usu- 
ally placed on lexical dependencies in statistical 
models one might have expected more improve- 
ment. Probably the poor performance is due 
in part to the fairly large differences between 
the parses from which the lexical dependencies 
were estimated and the parses produced by the 
LFG. LFG parses are very detailed, and many 
ambiguities depend on the precise grammatical 



Verbmobil corpus (324 sentences, 172 non-auxiliary features) 
Auxiliary features used Indistinguishable Correct - log PL 
(none) 9 180 401.3 

// 8 183 401.6 

flJcfn 8 180.5 404.0 

Homecentre corpus (481 sentences, 186 non-auxiliary features) 
Auxiliary features used Indistinguishable Correct - log PL 
(none) 45 283.25 580.6 

/; 34 284 580.6 

fiJcfn 34 285 582.2 

Table 1: The effect of adding auxiliary lexical dependency features to a SLFG. The auxiliary 
features are described in the text. The column labelled "indistinguishable" gives the number of 
indistinguishable sentences with respect to each feature set, while "correct" and "- log PL" give 
the correct parses and pseudo-likelihood measures respectively. 



relationship holding between a predicate and its 
argument. It could also be that better perfor- 
mance could be achieved if the lexical dependen- 
cies were estimated from a corpus more closely 
related to the actual test corpus. For example, 
the verb feed in the Homecentre corpus is used in 
the sense of "insert (paper into printer)" , which 
hardly seems to be a prototypical usage. 

Note that overall system performance is quite 
good; taking the unambiguous sentences into 
account the combined LFG parser and statisti- 
cal model finds the correct parse for 73% of the 
Verbmobil test sentences and 80% of the Home- 
centre test sentences. On just the ambiguous 
sentences, our system selects the correct parse 
for 56% of the Verbmobil test sentences and 59% 
of the Homecentre test sentences. 

6 Conclusion 

This paper has presented a method for incorpo- 
rating auxiliary distributional information gath- 
ered by other means possibly from other corpora 
into a Stochastic "Unification-based" Grammar 
(SUBG). This permits one to incorporate de- 
pendencies into a SUBG which probably can- 
not be estimated directly from the small UBG 
parsed corpora available today. It has the virtue 
that it can incorporate several auxiliary dis- 
tributions simultaneously, and because it asso- 
ciates each auxiliary distribution with its own 
"weight" parameter, it can scale the contribu- 
tions of each auxiliary distribution toward the 
final estimated distribution, or even ignore it 



entirely. We have applied this to incorporate 
lexical selectional preference information into 
a Stochastic Lexical-Functional Grammar, but 
the technique generalizes to stochastic versions 
of HPSGs, categorial grammars and transfor- 
mational grammars. An obvioTis extension of 
this work, which we hope will be persued in the 
future, is to apply these techniques in broad- 
coverage feature-based TAG parsers. 
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